Text mining for large medical text datasets and corresponding medical text classification using informative feature selection

ABSTRACT

Techniques include performing text mining on a set of case reports in text format to determine a set of grammar rules to be used to determine whether case reports meet a medical condition. The text mining includes performing feature selection, used to determine the set of grammar rules, that combines standardized case definitions with experience of medical officers for the medical condition and outputting the set of grammar rules. Another technique includes applying grammar rule(s) to new case report(s), the grammar rule(s) previously determined at least by performing text mining comprised of performing feature selection, used to determine the set of grammar rules, that combines standardized case definitions with experience of medical officers for the medical condition. Indication(s) are output of whether the new case report(s) meet or do not meet the medical condition. The techniques may be performed by a method, an apparatus, and a program product.

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims the benefit under 35 U.S.C. §119(e) ofU.S. Provisional Patent Application No. 61/651,210, filed on May 24,2012, the disclosure of which is hereby incorporated by reference in itsentirety.

BACKGROUND

This invention relates generally to text mining and, more specifically,relates to text mining in large medical text datasets.

This section is intended to provide a background or context to theinvention disclosed below. The description herein may include conceptsthat could be pursued, but are not necessarily ones that have beenpreviously conceived, implemented or described. Therefore, unlessotherwise explicitly indicated herein, what is described in this sectionis not prior art to the description in this application and is notadmitted to be prior art by inclusion in this section.

The following abbreviations that may be found in the specificationand/or the drawing figures are defined as follows:

AE Adverse Event

AEFI Adverse Events Following Immunizations

BOW Bag Of Words

EHR Electronic Health Record

MedDRA Medical Dictionary for Regulatory Activities

ML Machine Learning

MO Medical Officer

MR Medical Record

NLP Natural Language Processing

PT Preferred Term

SRS Spontaneous Reporting System

TC Text Classification

TM Text Mining

VAERS Vaccine Adverse Event Reporting System

Biomedical research is often confronted with large datasets containingvast amounts of free text that have remained largely untapped sources ofinformation. The analysis of these data sets poses unique challenges,particularly when the goal is knowledge discovery and real-timesurveillance. See Sinha et al., “Large datasets in biomedicine: adiscussion of salient analytic issues”, Journal of the American MedicalInformatics Association, 16(6):759-67 (2009). Spontaneous ReportingSystems (SRSs), such as the U.S. Vaccine Adverse Event Reporting System(VAERS), encounter this issue. See Singleton et al., “An overview of theVaccine Adverse Event Reporting System (VAERS) as a surveillancesystem”, Vaccine, 17(22):2908-17 (1999).

When extraordinary events occur, such as the H1N1 pandemic, routinemethods of safety surveillance struggle to produce timely results due tothe resource-intensive nature of the manual review. For instance,Medical Officers have to peruse these reporting systems and determinewhether adverse effects occur, e.g., as a result of the H1N1 vaccine.Consequently, there is an urgent need to develop alternative approachesthat facilitate efficient report review and identification of safetyissues resulting from the administration of vaccines. Textclassification (TC) provides an alternative and more efficient processby distinguishing the most relevant information from adverse event (AE)reports.

SUMMARY

Techniques are presented that include performing text mining on a set ofcase reports in text format to determine a set of grammar rules to beused to determine whether case reports meet a medical condition. Thetext mining includes performing feature selection, used to determine theset of grammar rules, that combines standardized case definitions withexperience of medical officers for the medical condition. The techniquesinclude outputting the set of grammar rules. The techniques may beperformed by a method, an apparatus, and a program product.

Additional techniques are presented that include applying one or moregrammar rules to one or more new case reports. The one or more grammarrules are previously determined at least by performing text miningcomprised of performing feature selection, used to determine the set ofgrammar rules, that combines standardized case definitions withexperience of medical officers for the medical condition. The additionaltechniques include outputting, based on the applying, one or moreindications of whether the one or more new case reports meet or do notmeet the medical condition. The techniques may be performed by a method,an apparatus, and a program product.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a block diagram of processing performed to determine a medicalcondition (e.g., anaphylaxis) based upon a medical text dataset (e.g.,VAERS);

FIG. 2 is a flowchart of a method for text mining for large medical textdatasets and corresponding medical text classification using informativefeature selection;

FIG. 3 is a set of grammar rules used by a rule-based classifier toclassify an entry in the VAERS as anaphylaxis;

FIG. 4 presents an example of text mining results using an exemplaryembodiment of the instant invention;

FIG. 5 is Table 1, illustrating macro-averaged metrics for MLclassifiers' performance on the first feature representation (lemmas),where the ranks of classifiers are included in parentheses, and theevaluation measures and classifiers are as follows: R: Recall; P:Precision; F: F-measure; NB: Naïve Bayes; ME: Maximum Entropy; DT:Decision Trees; RPCT: Recursive Partitioning Classification Trees; BT:Boosted Trees; w-SVM: Weighted Support Vector Machines; s-SVM: SVM forSparse Data; SB: Stochastic Boosting; MARS: Multivariate AdaptiveRegression Splines; RDA: Regularized Discriminant Analysis; RF: RandomForests; GAM: Generalized Additive Model; w-kNN: Weighted k-NearestNeighbors;

FIG. 6 is Table 2, illustrating macro-averaged metrics for MLclassifiers' performance on the second feature representation (low-levelpatterns), where the ranks of classifiers are included in parentheses,where the evaluation measures and classifiers are as follows (it shouldbe noted that Friedman's test indicated no statistically significantdifferences between the classifiers): R: Recall; P: Precision; F:F-measure; NB: Naïve Bayes; ME: Maximum Entropy; DT: Decision Trees;RPCT: Recursive Partitioning Classification Trees; BT: Boosted Trees;w-SVM: Weighted Support Vector Machines; s-SVM: SVM for Sparse Data; SB:Stochastic Boosting; MARS: Multivariate Adaptive Regression Splines;RDA: Regularized Discriminant Analysis; RF: Random Forests; GAM:Generalized Additive Model; w-kNN: Weighted k-Nearest Neighbors;

FIG. 7 is Table 3, illustrating macro-averaged metrics for rule-basedclassifier's performance (high-level patterns), where misclassificationerror rate (MER) was also calculated, and where the evaluation measuresand the classifier were as follows: R: Recall; P: Precision; F:F-measure; rule-based.

FIG. 8 is Supplementary Table 4, illustrating average sensitivity(sens), average specificity (spec) and their associated standard errors(SE) of the best performing ML classifiers and the rule-based classifierover testing and validation sets; also, the corresponding averagepositive predictive value (PPV), and average negative predictive value(NPV) are provided, where the classifiers are as follows: BT: BoostedTrees; w-SVM: Weighted Support Vector Machines;

FIG. 9 is Table 5, illustrating mean misclassification error rate(mean-MER) over the testing and validation data sets and the associatedstandard error (SE) for the different ML classifiers in the case oflemmas and low-level patterns, where the classifiers are as follows: NB:Naïve Bayes; ME: Maximum Entropy; DT: Decision Trees; RPCT: RecursivePartitioning Classification Trees; BT: Boosted Trees; w-SVM: WeightedSupport Vector Machines; s-SVM: SVM for Sparse Data; SB: StochasticBoosting; MARS: Multivariate Adaptive Regression Splines; RDA:Regularized Discriminant Analysis; RF: Random Forests; GAM: GeneralizedAdditive Model; w-kNN: Weighted k-Nearest Neighbors; and

FIG. 10 is a block diagram of a system suitable for performing exemplaryembodiments of the instant invention.

DETAILED DESCRIPTION

As stated above, there is an urgent need to develop alternativeapproaches that facilitate efficient report review and identification ofsafety issues resulting from the administration of vaccines. Textclassification (TC) provides an alternative and more efficient processby distinguishing the most relevant information from adverse event (AE)reports.

Medical TC is the process of assigning labels to a span of text(sentence, paragraph, or document) using trained or rule-basedclassifiers. For trained or rule-based classifiers, see the following:Ambert et al., “A System for Classifying Disease Comorbidity Status fromMedical Discharge Summaries Using Automated Hotspot and Negated ConceptDetection”, Journal of the American Medical Informatics Association2009; 16(4):590-5; Cohen A M, “Five-way smoking status classificationusing text hot-spot identification and error-correcting output codes”,Journal of the American Medical Informatics Association 2008;15(1):32-5; Conway et al., “Classifying disease outbreak reports usingn-grams and semantic features”, International Journal of MedicalInformatics 2009; 78(12):47-58; Farkas et al., “Semi-automatedconstruction of decision rules to predict morbidities from clinicaltexts”, Journal of the American Medical Informatics Association 2009;16(4):601-5; Mishra et al., “A Rule-based Approach for IdentifyingObesity and Its Comorbidities in Medical Discharge Summaries”, Journalof the American Medical Informatics Association 2009; 16(4):576-9; Onget al., “Automated categorisation of clinical incident reports usingstatistical text classification”, Quality and Safety in Health Care 2010Aug. 19; doi:10.1136/qshc.2009.036657; Savova et al., “Mayo Clinic NLPsystem for patient smoking status identification”, Journal of theAmerican Medical Informatics Association 2008; 15(1):25-8; Solt et al.,“Semantic classification of diseases in discharge summaries using acontext-aware rule-based classifier”, Journal of the American MedicalInformatics Association 2009; 16(4):580-4. For both trained andrule-based classifiers, see the following: DeShazo et al., “Aninteractive and user-centered computer system to predict physician'sdisease judgments in discharge summaries”, Journal of BiomedicalInformatics 2010; 43(2):218-23; Yang et al., “A Text Mining Approach tothe Prediction of Disease Status from Clinical Discharge Summaries”,Journal of the American Medical Informatics Association 2009 July;16(4):596-600.

The utilization of Natural Language Processing (NLP) techniques mayprovide better classification results through improvements in textexploration. However, according to Cohen and Hersh, T C should be placedcloser to the Text Mining (TM) field than the full-blown NLP field. SeeCohen et al., “A survey of current work in biomedical text mining”,Briefings in Bioinformatics 2005 March; 6(1):57-71. TM and NLPtechniques have been used before to identify AEs in Electronic HealthRecords (EHR5). See the following: Hazlehurst et al., “Detectingpossible vaccine adverse events in clinical notes of the electronicmedical record”, Vaccine, 2009; 27(14):2077-83; Melton et al.,“Automated detection of adverse events using natural language processingof discharge summaries”, Journal of the American Medical InformaticsAssociation 2005; 12(4):448-57; Murff et al., “Electronically screeningdischarge summaries for adverse medical events”, Journal of the AmericanMedical Informatics Association 2003; 10(4):339-50; Wang et al., “Activecomputerized pharmacovigilance using natural language processing,statistics, and electronic health records: a feasibility study”, Journalof the American Medical Informatics Association 2009; 16(3):328-37.However, the issue of a complete surveillance system that could begeneralized has not been addressed yet.

Safety surveillance in VAERS (and other SRSs) has two main purposes. Thefirst purpose is monitoring known adverse effects for unusual featuresor increases in reporting rate (i.e., number of reports/number of doses)while looking for potential associations with new products (e.g., H1N1vaccine) or new demographic groups. The second purpose is looking forunexpected AEs by identifying unusual patterns. In the first case, weare more interested in the identification of the actual adverse cases,while in the second case we primarily need to know whether theidentified patterns represent “real” conditions in terms of clinicalsyndromes.

Here, a multi-level TM approach is presented that was applied to a groupof VAERS reports involving the AE of anaphylaxis for text classificationpurposes. This investigation of TM for anaphylaxis could serve as amodel for TM in the first purpose of safety surveillance in VAERS andcould also serve as the basis to generalize this work to other AEs thatare acute, serious, and occur in close temporal proximity to thevaccination. Our scope was not to present a fully developed system forAE identification but rather to study patterns in the narrative of VAERSreports that are used to identify a known adverse effect. The strengthof our study lies in demonstrating the feasibility of using TM on largeSRS databases to exploit the information content of a new data source,other than EHRs or clinical trials data, for adverse eventidentification as well as in saving time and human resources.

Concerning the Vaccine Adverse Event Reporting System (VAERS), this is apassive surveillance repository that monitors the number and type of AEsthat occur after the administration of vaccines licensed for use in theUnited States. See Varricchio et al., “Understanding vaccine safetyinformation from the vaccine adverse event reporting system”, ThePediatric Infectious Disease Journal 2004; 23(4):287-94. VAERS containsboth structured (e.g., vaccination date) and unstructured data (e.g.,symptom text). The VAERS case reports should be distinguished from anyother type of medical documentation (e.g., discharge summaries), sinceit is not only experts (physicians) but also non-experts (patients andrelatives) who act directly as the reporters of AEs. Therefore, specialprocessing is needed to handle the frequent non-medical syntax andsemantics.

Concerning a review process for the VAER, MOs review all serious anddeath VAERS reports manually. Specifically, the MOs review theunstructured free text fields to identify the clinical entities in agiven case report, decide upon the acquisition of additional information(e.g., request a copy of the medical records), and consider whether anyregulatory action is warranted. VAERS reports are coded with MedicalDictionary for Regulatory Activities (MedDRA) Preferred Terms (PTs). SeeBrown E. G., “Using MedDRA: implications for risk management”, DrugSafety 2004; 27(8):591-602. Non-medical data-entry personnel apply PTsto terms in AE reports according to coding conventions and algorithms;the codes are not considered to be medically confirmed diagnoses. MOsmay screen and select case reports based on MedDRA codes, but the MOscannot fully rely on the codes for the analysis of safety data, mainlydue to the MedDRA limitations. The inability of MedDRA to automaticallygroup PTs with similar meanings from different system organ classesmakes PT based searches incomplete unless the searches are based on avalidated Standardized MedDRA query, which are resource intensive todevelop. See Bousquet et al., “Appraisal of the MedDRA conceptualstructure for describing and grouping adverse drug reactions”, DrugSafety 2005; 28(1):19-34. This initial process is shown in FIG. 1, wherethe VAERS database 110 is subjected to PT and keyword search todetermine a number of flu reports 120.

The rest of the process is performed in two steps (see FIG. 1), both ofwhich are laborious and time-consuming. Step 1 involves manual review ofcase reports (in this example, flu reports 120), which can number in thethousands (e.g., 6034 in this example), while Step 2 involves (e.g.,manual) review of medical records (MRs) and other documentation for asmaller number of possible cases (see example for anaphylaxis; FIG. 1).The results of Step 1 are shown (classified reports 130) as 237potentially positive (pos) reports of a particular medical condition, inthis case anaphylaxis. Note that the results of Step 1 may also includethe 5,797 potentially negative (neg) reports of the particular medicalcondition. The results of Step 2 are shown (reference 140) as 100confirmed cases of the medical condition, in this example anaphylaxis.

Herein, a variety of TM techniques and automated classifiers areincorporated to reliably substitute the manual classification ofanaphylaxis case reports at the first step (Step 1) and, thus, reducehuman effort.

One technique for TM includes use of case definitions for medicalconditions. There are a number of case definitions for medicalconditions. The example of anaphylaxis will be used herein, but this isnot a limitation on the exemplary embodiments of the instant invention.

The Brighton Collaboration develops standardized, widely disseminated,and globally accepted case definitions for a large number of AdverseEvents Following Immunizations (AEFI). Each case definition is developedin a strict process that is monitored by a specific internationalworking group of up to 20 experts and, among others, incorporatessystematic literature search and the evaluation of the previousfindings. See Bonhoeffer, et al., “The Brighton Collaboration:addressing the need for standardized case definitions of adverse eventsfollowing immunization (AEFI)”, Vaccine 2002; 21(3-4):298-302. Based oncertain criteria, Brighton Collaboration defines the patterns thatshould be discovered in the reports of a surveillance system. Often, MOstry to match the patterns with the reported symptoms in each case report(or the medical record at Step 2 of FIG. 1). The Brighton Collaborationhas developed a case definition for anaphylaxis, which is an acutehypersensitivity reaction with multi-organ-system involvement and canrapidly progress to a life-threatening reaction. See Ruggeberg et al.,“Anaphylaxis: Case definition and guidelines for data collection,analysis, and presentation of immunization safety data”, Vaccine 2007Aug. 1; 25(31):5675-84. The case definition follows:

Brighton Collaboration Case Definition of Anaphylaxis 1. Major and MinorCriteria Used in the Case Definition of Anaphylaxis

Organ Systems Major Criteria Dermatologic or generalized urticaria(hives) or generalized erythema mucosal angioedema (Not hereditary),localized or generalized generalized pruritus with skin rashCardiovascular measured hypotension clinical diagnosis of uncompensatedshock, indicated by the combination of at least 3 of the following:tachycardia capillary refill time >3 sec reduced central pulse volumedecreased level or loss of consciousness Respiratory bilateral wheeze(bronchospasm) Stridor upper airway swelling (lip, tongue, throat,uvula, or larynx) respiratory distress - 2 or more of the following:tachypnoea increased use of accessory respiratory muscles(sternocleidomastoid, intercostals, etc.) recession cyanosis grunting

Organ Systems Minor Criteria Dermatologic generalized pruritus withoutskin rash or mucosal generalized prickle sensation localized injectionsite urticaria red and itchy eyes Cardiovascular reduced peripheralcirculation as indicated by the combination of at least 2 of:tachycardia and a capillary refill time of >3 sec without hypotension adecreased level of consciousness Respiratory persistent dry cough hoarsevoice difficulty breathing without wheeze or stridor sensation of throatclosure sneezing, rhinorrhea Gastrointestinal Diarrhoea abdominal painNausea Vomiting Laboratory Mast cell tryptase elevation > upper normallimit

For all levels of diagnostic certainty Anaphylaxis is a clinicalsyndrome characterized by: sudden onset AND rapid progression of signsand symptoms AND involving multiple (≧2) organ systems, as follows:Level 1 of diagnostic certainty ≧1 major dermatological AND ≧1 majorcardiovascular AND/OR ≧1 major respiratory criterion Level 2 ofdiagnostic certainty ≧1 major cardiovascular AND ≧1 major respiratorycriterion OR ≧1 major cardiovascular OR respiratory criterion AND ≧1minor criterion involving ≧1 different system (other than cardiovascularor respiratory systems) OR (≧1 major dermatologic) AND (≧1 minorcardiovascular AND/OR minor respiratory criterion) Level 3 of diagnosticcertainty ≧1 minor cardiovascular OR respiratory criterion AND ≧1 minorcriterion from each of ≧2 different systems/categories

Common causes for anaphylaxis include allergens, drugs, andimmunizations. See Ewan P W, “ABC of allergies: Anaphylaxis, BritishMedical Journal 1998; 316(7142):1442. According to the Brighton casedefinition for anaphylaxis, specific major and minor criteria aredescribed per organ system. MOs try to discover these criteria in eachcase report, fit them into a pattern, and classify the report asanaphylaxis or not. For example, when the MOs read the report“immediately after vaccination the patient presented with face oedema,difficulty breathing, red eyes, wheezing and localized rash at site ofinjection; also complained for weakness and reported fever two daysbefore vaccination” they classify the report as potentially positiveprimarily because there are at least two organ systems involved:dermatologic (face oedema, red eyes) and respiratory (difficultybreathing, wheezing). The described “rash” is localized and should notbe considered as a dermatologic criterion, while “fever” and “weakness”are neither related to the vaccination nor included in the casedefinition. It should be mentioned that the above narrative would alarmMOs even if the sudden onset (stated by “immediately after”) and theapparent rapid progression of symptoms (even though not clearly stated)were missing. Often, the time dimension is missing from VAERS reports,so MOs would still pick up this report for further review and definitionof the diagnostic certainty at Step 2 in FIG. 1.

A number of methods may be used to process the VAERS database 110 (e.g.,and other medical databases). For a corpus and validation set, a subsetwas selected of all the case reports that were submitted to VAERSfollowing influenza A (H1N1) 2009 monovalent vaccines (see Vellozzi, etal., “Adverse events following influenza A (H1N1) 2009 monovalentvaccines reported to the vaccine adverse events reporting system UnitedStates, Oct. 1, 2009-Jan. 31, 2010”, Vaccine 2010; 28(45):7248-55),covering a period from Nov. 22, 2009 to Jan. 31, 2010 (Ntotal=6,034 asshown in flu reports 120 of FIG. 1). This time-window corresponded tothe same period that a thorough analysis of H1N1 reports was performedfollowing the receipt of a safety signal from Canada in mid-November.See Varricchio et al., “Understanding vaccine safety information fromthe vaccine adverse event reporting system”, The Pediatric InfectiousDisease Journal 2004; 23(4):287-94; “Quality Investigation of Combo LotNumber A80CA007A of Arepanrix H1N1 (AS03-Adjuvanted H1N1 PandemicInfluenza Vaccine) in Canada”, Canadian Ministry of Health 2010 Mar. 12;and Reblin T., “REPANRIX™ H1N1 Vaccine Authorization for Sale andPost-Market Activities”, Canadian Ministry of Health, 2009 Nov. 12.Twelve MOs reviewed the reports daily (the workload share wasapproximately equal); their task was to use the Brighton Collaborationcriteria for anaphylaxis to label them as potentially positive,requiring further investigation, or negative. Then, in one session withall MOs participating, the potentially positive reports were selected byconsensus (Npos=237, as shown by the classified case reports 130 afterperformance of Step 1 of FIG. 1); the remaining reports 130 wereclassified as negative (Nneg=5,797). The classification (either positiveor negative) by the MOs was the gold standard for the study.

FIG. 2 is a flowchart of a method for text mining for large medical textdatasets and corresponding medical text classification using informativefeature selection. The blocks of FIG. 2 may be method operations,operations performed by hardware (e.g., by one or more processorsconfigured by execution of computer readable code to cause an apparatusto perform the operations, or hardware such as logic and/or anintegrated circuit configured to perform the operations), or operationsperformed by an apparatus in response to computer readable program codeconfigured to cause the apparatus to perform the operations. As shown inFIG. 2, the classified case reports 130 are used in certain blocks inFIG. 2.

Subsequently, an identification number and a symptom text field wereextracted (see block 205 of FIG. 2), and a class label was assigned(block 210) to each case report according to the gold standard (e.g.,‘pos’ vs. ‘neg’ label for potentially positive versus negative reports,respectively). These data were included in a text file (e.g., one textfile per report); all text files were further organized in a categorizedcorpus under two distinct categories (‘pos’ vs. ‘neg’). The corpus inthis example is classified case reports 130. Moreover, a second set (seevalidation set 201 of classified case reports) was created forvalidation purposes following a similar process: two MOs reviewedretrospectively (Feb. 1, 2010 to Apr. 28, 2010) the case reports forH1N1 vaccine and created a validation set (Nvalid=1,100) with the samedistributional properties as the original set, i.e., 4% (four percent)of the case reports were potentially positive for anaphylaxis(NvalidPos=44).

Regarding feature extraction, an aspect of the study was the TM (block213) of VAERS case reports. The starting point was the ‘informativefeature selection’ (see block 215), that is, the combination of theBrighton Collaboration criteria with the MOs' experience. Three featurerepresentations were used:

1. Important keywords (first feature representation=>keywords). Seeblock 220, where important keywords are represented as features.

2. Major and minor criteria that included one or more of the abovekeywords. For instance, MOs considered ‘epinephrine’ to be equal to amajor criterion. Major and minor criteria may be marked by the Brightoncriteria or by the MOs. Adding the feature representing the diagnosis ofanaphylaxis (sometimes stated in a case report) the feature space couldbe treated as a set of low-level patterns (second featurerepresentation=>low-level patterns). See block 220, where low levelpatterns are determined that have major and minor criteria that includedone or more of the keywords.

3. Filtering patterns that consisted of the above criteria (e.g.,‘pattern1’, ‘pattern2’, and ‘pattern3’; at least two major criteria, onemajor and one minor criterion, and three minor criteria, respectively).Diagnosis of anaphylaxis was also considered to be a filtering patternalone (‘pattern4’) and, thus, played a dual role in this work. It shouldbe mentioned that the proportion of cases that were detected based onthe explicit diagnosis of anaphylaxis were equal to 9% (16 out of 178)in the training set, 3% (2 out of 59) in the testing set and 9% (4 outof 44) in the validation set. It is noted that the symbol “%” means“percent”. All filtering patterns were treated as high-level patterns(third feature representation=>high-level patterns). See block 230,where high level (e.g., filtering) patterns are created that compriseone or more of the criteria (e.g., two major criteria, one major and oneminor criterion, or three minor criteria).

In further detail of the preceding description, for text miningprocesses and a rule-based classifier, first, the free text in thecorpus was processed using the appropriate NLP methods. Second, twodirections were pursued by: (i) creating the list of lemmas (hereaftercalled dictionary) to represent the keywords of interest (first featurerepresentation=>keyword=>lemma) (block 220), and (ii) developing theanaphylaxis lexicon, building the grammar, tagging and parsing the freetext. The grammar rules supported the extraction of major and minorcriteria (second feature representation (block 225) and patterns (thirdfeature representation (block 230) from the case reports. Using thesepatterns, the corresponding part of the algorithm (i.e., the rule-basedclassifier) classified the reports into potentially positive andnegative.

The technical details of these processes are now presented in furtherdetail. In particular, an exemplary text mining is now described.

The free text in the corpus was first normalized by converting the freetext to lowercase and then by removing the punctuation marks and thestop words. The free text was also tokenized without altering the wordorder in the constructed list. These operations occur before block 220.From this point on, two directions were pursued, guided primarily by thepredefined output, i.e., the aforementioned feature representations.

First, a list was created of 43 lemmas (called a dictionary) torepresent the keywords of interest that were either identified in theBrighton case definition (see above) or suggested by the MOs as add-onsto Brighton guidelines (first feature representation=>keyword=>lemma(block 220)). Lemmas were selected to handle the disjunctions ofkeywords; for example ‘swell’ represented the disjunction: ‘swelled’ or‘swelling’ or ‘swells’ or ‘swollen’. Based on this principle, each casereport was initially treated as a bag-of-words (BOWs). The affixes wereremoved from all BOWs by using the Porter Stemmer. For the PorterStemmer, see Porter M F, “An algorithm for suffix stripping”, Program1980; 14(3):130-7. The stemming process could not handle alldisjunctions, so this process was adjusted to retain only one lemma perdisjunction, e.g., ‘face’ for both ‘face’ and ‘facial’ since PorterStemmer returned ‘face’ and ‘facial’, respectively.

The second direction included the anaphylaxis lexicon development, thegrammar building, the tagging, and the parsing of free text. The lexiconincluded the lemmas (similar to those included in the dictionary) alongwith their semantic tags, which described the broad category into whicheach lemma fell; for example, ‘neck’, ‘lip’, ‘tongue’ ‘uvula’, and‘larynx’ were tagged as ‘ANATOMY’. However, a more specific tag waspreferred for the majority of lemmas, following the format‘subcategory_CATEGORY’, e.g. ‘throat_ANATOMY’ for ‘throat’ and‘major_RESPIRATORY’ for ‘wheez’, ‘bronchospasm’, ‘stridor’, and‘distress’. This format supported the grammar structure adequately giventhat a lemma could fit into more than one grammar rule or even representa major or minor criterion itself (See FIG. 3). Any out-of-lexicon itemsas well as numbers were tagged as ‘UNIMPORTANT’ by the semantic tagger.Also, the lexicon handled a few abbreviations, such as ‘sob’ (stands for‘shortness of breath’).

The grammar rules supported the extraction of major and minor criteriafrom the case reports and the subsequent definition of filteringpatterns (block 230). To accomplish these tasks, the grammar rules 235were divided as follows (see also FIG. 3, which is a set of grammarrules used by a rule-based classifier to classify an entry in the VAERSas anaphylaxis) (Note that these rules are basically the patterns thatwere found in blocks 220/225/230):

i. One supporting rule 310 representing lemmas or numbers that weretagged as ‘UNIMPORTANT’; this rule was formed by one element only.

ii. Eight basic rules 320 representing the major and minor criteria, aswell as the diagnosis of anaphylaxis; these rules were formed by one ormore ‘important’ lemmas and optional repetitions of the supporting rule.

iii. One excluding rule 330 representing the ‘important’ lemmas thatfailed to fit into any of the basic rules; this rule was formed by oneelement only.

iv. Four advanced rules 340 representing the filtering patterns; theserules were formed by two or three basic rules and optional repetitionsof the excluding rule.

Python (version 2.6.4) was the tool used for both the TM of case reportsand the development of the rule-based classifier. The block 213therefore produces a set of grammar rules 235.

Concerning the patterns, an example is now presented with pattern1.pattern1: <MCDV><exclude>*<MDERM> contains three nested rules asfollows:

1-MCDV rule: this rule returns any major cardiovascular symptom (as thisis defined in the Brighton collaboration case definition);

2-MDERM rule: this rule returns any major dermatological symptom (asthis is defined in the Brighton collaboration case definition);

3-exclude rule: this rule collects all the “garbage”, i.e. those tokens(e.g., words) that form a criterion only as a part of a multi-token; forexample, the token “throat” means nothing if it does not come with“swollen”. Therefore, this rule allows the MCVD and MDERM to be combinedand equate to pattern 1 (or diagnostic level 1 as per BrightonCollaboration definition).

The * (asterisk) means that there might be 0 (zero) or a number of“exclude” occurrences between the symptoms of interest. The position ofthe * indicates where these occurrences might be located.

All other patterns have the same interpretation as above.

The grammar rules 235 may be applied (e.g., using a rule-basedclassifier) to new case reports (e.g., as embodied in the validationsset 201 of classified case reports, although the classification is notused) to classify case reports as positive or negative. The classifiedcase reports are output (e.g., to a file in memory). See block 235. Inblock 245, the system can compare the rule-based classification with theclassification in validation set.

A number of supervised machine learning (ML) classifiers that have beenpreviously found to be the appropriate solutions for TC problems weretrained. Most of these have been also used for medical TC (binary ornot). The ML classifiers were as follows: Naïve Bayes (NB) (5), MaximumEntropy (ME) (27), Decision Trees (DT) (6), Recursive PartitioningClassification Trees (RPCT) (28), Boosted Trees (BT) (29), WeightedSupport Vector Machines (w-SVM) (3;4), SVM for Sparse Data (s-SVM) (30),Stochastic Boosting (SB) (31), Multivariate Adaptive Regression Splines(MARS) (32), Regularized Discriminant Analysis (RDA) (33), RandomForests (RF) (34), Generalized Additive Model (GAM) (31), and Weightedk-Nearest Neighbors (w-kNN) (35). The splitting rules that were used bythe decision tree classifiers (DT, RPCT and BT) should not be confusedwith the advanced grammar rules that represented the filtering patternsand were used by the rule-based classifier, as it was described above.For this reason, the decision tree classifiers and the rule-basedclassifier could not form a homogeneous group in a hypotheticalcomparison with the other classifiers.

An issue with ML classifiers is that these classifiers assign equalweights to all classes, no matter their rarity; this may affect theirperformance in favor of the commonest class. Thus, we included a weightparameter into the training process that allowed us to handle theproblem of class imbalance; for instance, libSVM tool (library for SVM)allows the addition of the weight parameter to the training process(36). The weight for each class was calculated by the formula proposedby Cohen (4):

w _(class)=(N _(total) −N _(class))/N _(total),

where Ntotal is the total number of cases and Nclass the number of casesper class. Our weighted approach was also applied to BT, GAM, and s-SVM.

Python (version 2.6.4), several packages in R-statistics (version2.11.1), and libSVM tool were used for the training of binaryclassifiers, as well as for the calculation of metric values in thetesting and validation sets.

Concerning evaluation metrics and statistical analysis, for evaluatingthe performance of class-labeling by the classifiers, themacro-averaging of standard recall (R→macro-R), precision (P→macro-P),and F-measure (F→macro-F) were used. macro-R and macro-P are preferablehere since they are dominated by the more rare ‘pos’ category (37).macro-P and macro-R were analyzed using Friedman's test that avoids thenormality assumption and analyzes the ranks of classifiers within thedata sets (38). Moreover, the test is appropriate for use with dependentobservations, which is the case here, because classifiers were testedusing the same data sets. The original level of significance of 0.05 wasadjusted by Bonferroni to account for multiplicity in testing (thus, thelevel of the test was 0.0125, adjusted further for multiple comparisonsto 0.001).

An exploratory error analysis was performed by presenting the meanmisclassification error rate (mean-MER) for each classifier over thedata set. Each classifier's SE is computed first on each data set(testing or validation) using the formula:

$\left\lbrack \frac{{\hat{p}}_{ij}\left( {1 - {\hat{p}}_{ij}} \right)}{n_{i}} \right\rbrack^{1/2},$

where n, i=1,2 is 1508 or 1100 respectively and {circumflex over(p)}_(ij), j=1, . . . , 13 is the error rate of the classifier on thetesting and validation set. The mean-MER is obtained by averaging theindividual data set dependent error rates and the associated SE iscomputed using a formula that adjusts for the different sizes of thedata sets.

Furthermore, the impact of specific features on the classification wasevaluated by computing the information gain for each unique lemma, low-and high-level pattern; information gain is employed as a term-goodnesscriterion for category prediction and is based on the presence orabsence of a term in a document (39). The FSelector package inR-statistics (version 2.11.1) and the training set were used for thecorresponding calculations.

Results will now be described. For text mining results, the applicationof the instant TM techniques and the vectorization of the TM results areshown in FIG. 4. The same processes were applied to all reports (both inthe corpus and the validation set). The bag-of-lemmas (BOLs)representing a case report was compared against the dictionary and avector of binary values was constructed to indicate the presence orabsence of dictionary entries in the case report; the vector (hereaftercalled type I vector) was extended by one position to include the classlabel of the report. The whole process is presented in the left branchof FIG. 4.

The semantic tagger assigned tags to the lemmas of each case report,while the parser interpreted the grammar to fit each tagged lemma into arule; the rules were executed in order. Thus, the parsing processreturned the low- and high-level patterns (FIG. 4, right branch). Again,a binary vector was created to indicate the presence or absence of thelow-level patterns in each class-labeled report; each vector (hereaftercalled type II vector) had nine positions corresponding to the eightlow-level patterns plus the class label. As for the high-level patternsno vectorization was performed since the rule-based classifier processedthis output directly by classifying a case report as potentiallypositive when the parser output matched any of ‘pattern1’, ‘pattern2’,‘pattern3’, or ‘pattern4’; the report was otherwise classified asnegative. The rule-based classification was straightforward and wasapplied to the case reports of all sets without getting into anytraining process.

On the other hand, in order to examine ML classifiers, all vectors (bothtype I and type II) of the original set were randomly split into atraining and a testing set following the 75%-25% splitting rule(Ntrain=4,526 and Ntest=1,508, respectively). Both sets had the samedistributional properties, i.e. 4% of the case reports were potentiallypositive for anaphylaxis. The performance of rule-based and MLclassifiers over the testing and validation sets is presented in Tables1-3 (shown in FIGS. 5-7). Supplementary Table 4 (FIG. 8) presents theaverage (over testing and validation sets) sensitivity and specificityand their associated standard errors, as well as the positive andnegative predictive value of the best (across metrics) performingmachine learning classifiers and rule-based classifier. Note here theequivalent performance of these classifiers in terms of all metricspresented. Yet, the rule-based classifier saved considerablecomputational time since it did not have to be trained, a process thatwas followed for w-SVM and BT, i.e. the best ML classifiers. Moreover,the w-SVM and BT classifiers were also using features extracted from theBrighton Collaboration case definition and that, to a large extent,accounts for their good performance. Further analysis was performed formacro-averages and MER (see next paragraph).

Concerning quantitative and qualitative error analysis, the nullhypothesis of no difference, on average, among the ML classifiers interms of macro-R for lemmas was rejected by Friedman's test (teststatistic=12.37; p-value=0.000058, with F12,12 distribution). Bonferronicorrected multiple comparisons (a=0.0125) indicated that BT, w-SVMclassifiers performed best in terms of macro-R but were statisticallyequivalent to NB, RDA or GAM. Notice that macro-R is equivalent to afunction of sensitivity. MER analysis indicated that the mean-MERassociated with BT and w-SVM classifiers was 0.08 (SE=±0.0053) and0.1015 (SE=±0.006), respectively; this was 2.5 and 3.2 times higher thanthe smallest mean-MER among the equivalent classifiers (Table 5, FIG.9). Additionally, the higher variability of these two classifiers ascompared with the remaining classifiers should be noted. Therefore,higher macro-R comes at the expense of a higher mean-MER (and associatedSE). Similarly, the null hypothesis of no differences, on average, amongthe ML classifiers in terms of macro-P was rejected by Friedman's test(test statistic=4.2346; p-value=0.0092). Bonferroni, adjusted multiplecomparisons, indicated that the worst performance, in terms of macro-P,was achieved by BT and w-SVM classifiers, while GAM exhibited the bestperformance. GAM, RF, w-KNN, ME, DT, RPCT and s-SVM were statisticallyequivalent in terms of macro-P.

Similar results were obtained for low-level patterns. Specifically, BTand w-SVM classifiers performed best in terms of macro-R with NB, ME,RDA and GAM being statistically equivalent to BT and w-SVM. Our erroranalysis indicated higher mean-MER for the latter classifiers [for BTand w-SVM were 0.0815 (±0.0055) and 0.1015 (±0.0059), respectively].These rates were two and three times higher the minimum mean-MERobtained from the group of equivalent to the best performingclassifiers. In terms of macro-P, BT and w-SVM exhibited the worstperformance, while best performers were RF and MARS. Statisticallyequivalent to the latter were GAM, w-KNN, DT, ME, SB, s-SVM and RPCT.The error analysis attributed a mean-MER of 0.032 (SE: ±0.0035) to RFand MARS, which is the smallest among all.

For the rule-based classifiers the mean-MER over the testing andvalidation set was 0.0585 (SE=±0.00465). This error rate was smallerthan the best performing ML classifiers, while the performance in termsof macro-recall/precision was equivalent. Moreover, the mean macro-R was0.8690 (SE=±0.00674), the mean macro-P was 0.6875 (SE=±0.00919), and themean macro-F was 0.7675 (SE=±0.00839).

In addition to the aforementioned quantitative error analysis, weevaluated the classification performance of the ML classifiers using theArea Under the ROC Curve (AUC). There are several compelling reasons asto why the AUC is appropriate for quantifying the predictive ability ofclassifiers. One reason being the use of AUC for testing whetherpredictions are unrelated to true outcomes is equivalent to using theWilcoxon test. The best performing classifiers when AUC was used were BTand w-SVM. The AUC of BT when lemmas were used in the testing andvalidation set was, respectively, 0.9234 (95% Confidence Interval (CI)was [0.9198, 0.9270]) and 0.7888 (95% CI was [0.7771, 0.8006]). Thecorresponding values of the AUC and 95% CI when low-level patterns wereused for the BT classifier and for testing and validation set,respectively, were 0.8686 (95% CI was [0.8636, 0.8737]) and 0.8944 (95%CI was [0.8896, 0.8992]). For the w-SVM and for lemmas in testing andvalidation sets we obtained an AUC of 0.8706 and 0.8087, respectively,with corresponding 95% CI given as [0.8662, 0.8750] and [0.7989,0.8185]. These values indicate equivalent performance of BT and w-SVMclassifiers in terms of their predictive ability of identifying trulypositive reports.

We also performed a qualitative error analysis aiming at understandingif any patterns are present that allow the misclassification of reportsas negative when they are truly positive. The total number of falsenegative (FN) reports, in both testing and validation sets, returned bythe three best performing classifiers (rule-based, BT and w-SVM) was 36(16 and 20 in the testing and validation set, respectively). None ofthese reports contained the word “anaphylaxis” and all were missing anequivalent diagnosis. MOs examined, at a second stage, these 36 reportsand reclassified as truly positive 15 of them (2 and 13 in the testingand validation set, respectively). Recall that the total number of trulypositive reports in both sets equals 103. Our best performing rule-basedclassifier misclassified falsely as negative only 7 of these reports.The symptom text of the 15 reports and more details about theirqualitative error analysis are included below in a section entitled“Qualitative Error Analysis for Symptom Text”.

The feature analysis identified six lemmas (‘epinephrin’, ‘swell’,‘tight’, ‘throat’, ‘anaphylaxi’, ‘sob’) with very similar informationgain results. The diagnosis of anaphylaxis and epinephrine were amongthe most predictive features too, when they were treated as low-levelpatterns, along with major and minor respiratory and major dermatologiccriteria. Regarding high-level patterns all (‘pattern1’, ‘pattern2’, and‘pattern4’ that represented the diagnosis of anaphylaxis) wereequivalent in terms of information gain, excluding ‘pattern3’. Thelatter should be attributed to the extremely low (1.12%) or zerofrequency of ‘pattern3’ in the potentially positive and negative reportsof the training set, respectively. In all the other cases the frequencyof the most important predictive features was remarkably higher in thesubset of the potentially positive reports.

A discussion of the study now follows. In this study, we examined theeffectiveness of combining certain TM techniques with domain expertknowledge in the case of VAERS for TC purposes; to our knowledge, noprevious efforts have been reported for TM and medical TC in VAERS orany other SRS, despite the fact that various methods have been appliedbefore to other data sources showing the potential for AEidentification. For example, NLP methods have been applied to dischargehospital summaries (17) and other data mining methods to structured EHRdata (40-42). Our validated results showed that TM in any level couldeffectively support TC in VAERS. For example rule-based, BT and w-SVMclassifiers appeared to perform well in terms of macro-R, still withsome MER cost. A simple calculation over 10,000 reports for twoclassifiers (e.g. w-SVM and NB for low-level patterns) with MERs equalto 0.10 and 0.04, respectively, would show an actual difference of 600misclassified (either as potentially positive or as negative) reportsbetween them. The actual cost in terms of extra workload would be thosemisclassified as potentially positive (based on our data that would beequal to 494 reports) but the actual cost in terms of safetysurveillance would be those misclassified as negative (i.e. 106reports). Based on our error analysis, less than 7% (7 out of 103) ofthe true positive cases would be falsely classified as negative for ourbest performing classifier, i.e. the rule-based classifier. We believethat this level of misclassification, in the context of the extensiveknown limitations of SRS, is probably acceptable, although we hope toengage in future efforts to refine our algorithm to reduce this evenfurther. This further illustrates that one of the important propertiesof a classifier that is used to identify rare adverse events is highsensitivity because it returns a smaller number of falsely negativereports.

It could be argued that our approach lacks the automated featureextraction aspect, which has been previously reported as a strategy forTC (43). The issue of automatically extracting features thatcharacterize the AE accurately requires care. The problem we are calledto solve is the identification of rare or very rare events from the dataat hand. Because features need to be related not only statistically butalso causally to the outcome, informative feature selection betterserves our purposes. The basis for our claim has been the availabilityof solid standards (i.e. Brighton case definitions) that are being usedby physicians in their daily practice. Accordingly, the extraction ofthree feature representations supported the application of ourmulti-level approach. Thus, we treated the case reports not only as BOLslooking for lemmas (7) [Bag Of Words approach (stemmed or unstemmed) israther limited (13)] but also as sources of patterns (low- andhigh-level); we extracted these patterns to examine their role in TC.

Informative feature selection mandated not only the use of Brightoncriteria but also MOs' contributions since: i) certain criteria were notmet in the case reports and should be excluded from the feature space,e.g. ‘capillary refill time’, ii) non-medical words were often used bypatients to describe a symptom and should be included, e.g. the word‘funny’ within variations of the phrase ‘my throat felt funny’ todescribe ‘itchy throat’ or ‘throat closure’, and iii) other words raiseda concern for further investigation even though they were not listed inBrighton definitions, e.g. ‘epinephrine’ or ‘anaphylaxis’.

Regarding our TM methods, the construction of a controlled dictionaryand lexicon is considered laborious, demanding, and costly because itrelies on the recruitment of human experts (44). However, theinformative development of a flexible and relatively small controlleddictionary/lexicon appeared to be very effective in our study. The sameapplied to the use of the dedicated semantic tagger. A part-of-speechtagger would assign non-informative tags to a span of text (i.e. symptomtext in VAERS) that follows no common syntax; it would not support thegrammar rules either. The grammar was also built in the same context: tobetter serve the extraction of the feature representations andfacilitate both the rule-based classification and the training of MLclassifiers.

Rule-based TC systems have been criticized for the lack ofgeneralizability of their rules, a problem defined as ‘knowledgeacquisition bottleneck’ (44). However, their value in handling specificconditions should not be ignored, such as in the Obesity NLP Challenge,where the top ten solutions were rule-based (27). ML methods are not astransparent as the rule-based systems but have been used extensively forTC (44). Our results showed that the rule-based classifier performedslightly better, probably due to the informative feature selection.Either rule-based or ML techniques could be applied to SRS databases andallow better use of human resources by reducing MOs' workload.

It could be argued that ensembles or a cascade of classifiers or even amodified feature space would handle the classification errors.Nevertheless, the principles of our study and the nature of VAERS wouldrequire the consideration of certain aspects prior to the application ofsuch strategies. First, the construction of the feature space was basedon the domain expert contribution; any alterations (use of new lemmas orintroduction of new rules) should be approved accordingly to be valid.Second, a classification error will be always introduced by the MedicalOfficers who decide to acquire more information for a ‘suspicious’report even if it does not meet all the criteria.

The methodology that was described in this paper and the discussion ofthe related aspects raises the interesting question of generalizability,i.e. the transfer of components to the identification of other AEFIs.The development of a broader medical lexicon and a set of basic rulescould be suggested to support the extraction of all symptoms related tothe main AEFIs, such as the Guillain-Barre Syndrome (GBS) and the AcuteDisseminated Encephalomyelitis (ADEM). Based on these key components,other advanced rules representing the specific criteria per AEFI (asstated in Brighton definitions and described by MOs) could classify eachreport accordingly.

Our study lies partly in the field of text filtering since itinvestigated ways to automate the classification of streams of reportssubmitted in an asynchronous way (45). Generally, MOs' intention istwo-fold: first, to identify the potentially positive and block thenegative reports (step 1); second, to further classify those that provedto be positive into more specific categories (step 2), e.g., anaphylaxiscase reports into levels of diagnostic certainty. This process issimilar to the classification of incoming emails as ‘spam’ or ‘non-spam’and the subsequent categorization of the ‘non-spam’ e-mails (46-48).Here, any further categorization would require the information gatheringthrough the review of medical records that are provided in portabledocument format (pdf) only. The inherent difficulties related to theproduction of these files limit their usability and the possibility ofutilizing this source remains to be investigated.

To conclude, our study demonstrated that it is possible to apply TMstrategies on VAERS for TC purposes based on informative featureselection; rule-based and certain ML classifiers performed well in thiscontext. It may be possible to extend the current work and to apply thesame TM strategy regarding other AEs by incorporating experts' input ina semi-automated feature extraction framework.

FIG. 10 is an exemplary system suitable for performing exemplaryembodiments of the invention. This system comprises a computer system1000 comprising one or more processors 1010, one or more memories 1020(comprising computer program code 1030), and one or more user inputinterfaces 1060 (e.g., network interfaces, touchscreen interfaces, mouseinterfaces, keyboard interfaces, and the like). This example shows adisplay 1040 that also includes a user interface 1050. The userinterface could show information such as the potentially positivereports or any other information described above that might be desiredto be shown to a user. The computer readable code 1030, when executed bythe one or more processors 1010, causes the computer system 1000 toperform the operations described herein.

Qualitative Error Analysis for Symptom Text

This section contains symptom text of the 15 reports and more detailsabout their qualitative error analysis. The small number of FNreclassified reports does not allow the definition of firmmisclassification patterns for any of the three classifiers.Interestingly, five reports (Reports 1-5, excluding BT for low-levelpatterns in the validation set; Supplementary Table 6) weremisclassified by the three algorithms for all feature representations;these cases were picked as possible anaphylaxis based on the generalcontext of the report and the MOs' concern for accessing moreinformation in step 2. In five other cases (Reports 6-10; SupplementaryTable 6) both w-SVM and BT classifiers failed to classify them aspossible anaphylaxis using lemmas; the same was observed in three othercases (Reports 11-13; Supplementary Table 6) either for w-SVM or BTalone. On the other hand, the same reports were correctly classified bythe three classifiers when using low- or high-level patterns. Theremaining two reports were incorrectly misclassified by either therule-based or all three classifiers using high-level (Report 14;Supplementary Table 6) and low- and high-level (Report 15; SupplementaryTable 6) patterns, respectively; here, the existing elements couldsupport the conceptual definition of high- and low-level patterns,however the text structure did not facilitate that for the rule-basedclassifier. The above observations showed that: (1) the low- andhigh-level patterns better supported the identification of a potentiallypositive anaphylaxis reports for w-SVM, BT and rule-based classifiersand (2) the falsely negatively classified reports after thereclassification could be explained by either the MOs concern for moreinformation or the individual's reporting style that hid the existinganaphylaxis patterns.

SUPPLEMENTARY TABLE 6 Symptom text for the 15 reports of the testing andvalidation set that were reclassified as truly positive: Report SymptomText 1 Pt. experienced “extreme pain” at the time of injection thatradiated to her neck, back and chest. She became dizzy and nauseous 1hour after injection, she developed hives all over her body -experienced “deep muscle pain” in her legs 5 hours after, she also c/ofatigue and an uncomfortable chest sensation. Leg pain is worsening andheadache for 2 days. 2 Next day around 1pm experienced swelling in faceand itchyness, hives in abdomen which was relieved by Benadryl. 3 Hands,arms itchy-raised hives hands, neck itchy, resp difficulty. 10:15 Momnotified of transport. Transported via rescue. 4 pt given H1N1 vaccine20 min later she felt itchy, arm turning red, felt tongue enlarging. ptgiven 25 mg Benadryl 25 mg im, pt improved sent home on prn Benadryl andprednisone burst pack 5 After about 3-4 hours mother brought pt tovoluntary ER. He developed generalized urticaria, he was complaining ofdifficulty breathing. Patient hx of allergic reaction (pluria rash) toPCN and AMOX. No above meds at the time. Rx BENADRYL, prednisone p.o.PEPCID too. 6 PT. c/o of pains to stomach and H/R - PT. did vomitapprox. 1045. 1046 c/o hives, hives noted on bilateral forearms - 50 mgBENADRYL PO - vomited up. 7 Acute SOB and hives post vaccination. 8 Redskin, Itchy, Bright Red Blotches on neck, torso and groin, TroubleBreathing 9 Hives, puffy eyes, shortness of breath, hoarsey voice. 10 Istarted to have diffuclty breathing and started having Hives spreadabout my body 11 Throat became swollen and painful, vomited, stomachpains. The following information was obtained through follow-up and/orprovided by the government. 12/16/2009 PCP records for 12/8 and12/14/2009, patient with c/o's vomiting, mouth soreness, tongue swellingand urinary incontinence, tx'd with benadryl, was reported that childwas ill prior to vaccine 12 hives, itching, red and hot face, a littledifficulty breathing 13 Received H1N1 nasal~AM 12/29/2009. Taken to ERlater that evening with hives and wheezing. No hx RAD no probs withseasonal flu mist & mom discussed this w/ me at her physical on 2/5/10.14 About 10 mins after vaccines administered pt began to have troublebreathing, tongue swelling, wheezing. 15 Swelling of the face, throat,and eyes. Hives, rashes, and itching on day 2

Supplementary Table 7 includes the symptom text for the reports thatwere re-reviewed and classified as truly negative. The commoncharacteristic of these reports is that the criteria for anaphylaxiswere not fully met. Originally, MOs decided to classify them as positivefor two main reasons: (1) they were extremely cautious to protect thesafety of the public after receiving the signal from Canada and (2)based on that they wanted to acquire more information for any reportthat did not fulfill all the criteria but looked suspicious. However, inorder to perform an accurate qualitative error analysis, MOs were askedto re-review the 36 falsely negatively classified reports following onlythe Brighton criteria for anaphylaxis. Consequently, our top classifiersappeared to have correctly classified these reports (Supplementary Table7) as negative.

SUPPLEMENTARY TABLE 7 Symptom text for the 21 reports of the testing andvalidation set that were reclassified as truly negative: Report SymptomText 16 Swollen tongue, difficulty breathing. The following informationwas obtained through follow-up and/or provided by the government.1/29/10 Received medical information from patient: nausea. 1/28 and1/29/2010 MR and DC summary for 1/18-1/20/2010 prior to vaccineadministration and ED record for 1/26/2010 Dx Acute bronchitis patientwith c/o's cold sx and productive cough 17 Received H1N1 on 1/26/10 @15:45. 1/27/10 about 1000 developed headache, fatigue congestion, sorethroat. 1/28/10 afternoon, red puffy under eyes and hives front and backof trunk. Called HCP who advised BENADRYL. Hives resolved by 1/31/10.18 1. Dizziness. 2. Nausea Vomiting. 3. Hypertension. 4. Eyeitchiness/redness/weakness. Treatment Drugs Used: 1-Diphenhydramine2-Clonidine 19 Difficulty breathing after receiving seasonal flu mistand H1N1 shot. Required 3x hospital visits. Received 2x breathingtreatments, prescribed rescue inhaler and steriods for broch. spasms11/24 and 12/4/2009 ED records for 11/18/2009, Dx acute Bronchitispatient with c/o's SOB, chest pain. Tx: resp tx's with nebs 20 Increasedheart beat and an odd cough - lasted about ½ hr but cough went intoasthma and lasted about 1 week and have never had a reaction to any fluvaccine. 21 11/21/09 - Patient received H1N1 vaccination. While exitingfacility, developed chest tightness, SOB, lightheaded, cyanotic lips andnail beds. BP 122/64-88- 16. Patient remained seated until leavingfacility via ambulance to local Emergency Dept. No resolution insymptoms. Work up in ED was negative. 12/7 & 12/8/09: ED Recordsrecieved for dates of service 11/21/09. Dx: Probable vagal reaction,acute chest pain. Assessment: Presented after H1N1 vaccine with c/ochest tightness, tingling, anxiety, lightheadedness, SOB, cyanosis ofnail beds and lips. SaO2 94%. Medicated with Ativan 0.5 mg. Pulse oxincreased to 100% on room air. Felt well to go home. 22 Pt c/o hives,redness, and abdominal pain, 25 mg of PO Benadryl given. Update reportinformation to include: Adverse evernt - Adominal pain right side.Shortness of breath. Sleepy. Mom did not take patient to pediatrician.23 Emp. received H1N1 vaccine. States that approx 10 min later she beganto feel hot and her palms became sweaty and she vomited x 2. 24Headache, dry mouth, swollen eyes, rapid pulse, high blood pressure. Thefollowing information was obtained through follow-up and/or provided bythe government. MR received 01/11/10 for DOS 12/16/09. Pt was not seenby PCP or ER/Hosp and headaches resolved after 2 days. 25 Headache -nausea all Hives 25 mg BENADRYL oral given w/good recovery 26 1 c/othroat closing 2 transported by rescue to hospital The followinginformation was obtained through follow-up and/or provided by thegovernment. ED notes received 01/31/10 for DOS 11/02/09. DX: Allergicreaction. Pt presented with throat closing. Pt discharged home. 27 Thefollowing occurred two and half hours after receiving H1N1 shot12/15/09. Difficulty breathing, rapid heart beat, the patient neverexperienced any heart problems prior to the vaccine. 12/31/09, 1/4/10,1/6/10 Records received for ED, Hospital records and cardiology consultfor dates of service 12/15/09 to 12/28/09. Dx: Atrial fibrillation withrapid ventricular rate. Two hours after H1N1 vaccine experienced chesttightness, SOB and palpitations. Took Benadryl and presented to ED inatrial fibrillation with ventricular rate of 147. Given Cardizem bolusand then drip. Experienced improvement of sx. of chest tightness, SOB,palpitations and control of heart rate. Successfully cardioverted andstarted on heparin. Discharged on coumadin and Lovenox. Followed up on12/28/09 for persistent hypertension. Norvasc added to regimen. 28Student c/o itching and warmth to L arm along with pain after H1N1injection. Ice applied and mom notified. VS monitored. Cough began andstudent placed in supine position with feet elevated. BENADRYL givenafter itching and warmth noted. C/O dizziness - continued to monitor VS.C/O CP and nausea. The following information was obtained throughfollow-up and/or provided by the government. 1/7/2010 MD records for12/15/2009 patient with c/o's lt arm itching and redness, dizziness,chest pain and nausea tx'd with Benadryl 29 Pt called office 1/12/10 AMto report that she had developed following symptoms last night @ 19:30:throat swelling, chest and back pain, blurry vision, irreg rapid heartrate, cough. She did not seek med. attention. Did not takeantihistamines. All sx resolved without intervention. 30 Immediatelybecame pale, dizzy, with tingling in throat, difficulty speaking, mind“fuzzy”, difficulty hearing, nauseaus, difficulty breathing. Calleddoctor, took Benedryl, went with parents to ER where she received IVfluids, anti-nausea medications, steriods. Stayed in ER for 6 hours,continuing to be tired, nauseaus, dizzy 9 days later. The followinginformation was obtained through follow-up and/or provided by thegovernment. 1/25 and 1/29/2010 PCP notes, ED records for 1/9/2010, Dxadverse drug reaction, altered level of consciousness patient with c/o'sdizziness, altered level of consciousness, difficulty speaking, pale andlethargic, tx IVF, IV Solu-Medrol 31 mild muscle pain on 15th, on 16thmoderadte muscle pain, and on 17th and 18th sever muscle pain. Tylenoldid not help. 3 200 mg ibuprofen helped about 50%. 32 Swelling of Throatapprox. 5 hrs after vaccine admin. Fever day after vaccine admin.Hoarse, sore throat. Took diphenhydramine for swelling, ibuprofen forfever. 33 1 - 10 min post injection - SOB/pale, bilateral wheeze. 2 -albuterol rescue inhaler × 2 - poor result 3 - 11:25 o2 viamask/albuterol neb × 1/911 to hosp 1/21/10 = TC “panic” attack re:parent. No problems residual pt. recovered 34 From the initial injectionthere was a problem. It didnt come out the needle at first and had to betried again, pushing harder. It bunred immediately. Shortly after that,I began having stomach pains; heaviness in my arm where the injectionsite was; tingling in my fingers on that arm and a taste of metal in mymouth. I went to the walk in emergency room and the receptionist notedthat my throat, neck and chest were red. When the doctor examined me, healso noted that I had a bad reaction to it because my arm was swollenand puffy. 35 Per pt had shortness of breath and itching. 36 high fever(104) Difficulty breathing fast heart beat Cold Hands Cold feet PalenessBlue lips Wheezing red eyes weakness runny nose dizziness

As will be appreciated by one skilled in the art, aspects of the presentinvention may be embodied as a system, method or computer programproduct. Accordingly, aspects of the present invention may take the formof an entirely hardware embodiment, an entirely software embodiment(including firmware, resident software, micro-code, etc.) or anembodiment combining software and hardware aspects that may allgenerally be referred to herein as a “circuit,” “module” or “system.”Furthermore, aspects of the present invention may take the form of acomputer program product embodied in one or more computer readablemedium(s) having computer readable program code embodied thereon.

Any combination of one or more computer readable medium(s) may beutilized. The computer readable medium may be a computer readable signalmedium or a computer readable storage medium. A computer readablestorage medium may be, for example, but not limited to, an electronic,magnetic, optical, electromagnetic, infrared, or semiconductor system,apparatus, or device, or any suitable combination of the foregoing. Morespecific examples (a non-exhaustive list) of the computer readablestorage medium would include the following: an electrical connectionhaving one or more wires, a portable computer diskette, a hard disk, arandom access memory (RAM), a read-only memory (ROM), an erasableprogrammable read-only memory (EPROM or Flash memory), an optical fiber,a portable compact disc read-only memory (CD-ROM), an optical storagedevice, a magnetic storage device, or any suitable combination of theforegoing. In the context of this document, a computer readable storagemedium may be any tangible medium that can contain, or store a programfor use by or in connection with an instruction execution system,apparatus, or device.

A computer readable signal medium may include a propagated data signalwith computer readable program code embodied therein, for example, inbaseband or as part of a carrier wave. Such a propagated signal may takeany of a variety of forms, including, but not limited to,electro-magnetic, optical, or any suitable combination thereof. Acomputer readable signal medium may be any computer readable medium thatis not a computer readable storage medium and that can communicate,propagate, or transport a program for use by or in connection with aninstruction execution system, apparatus, or device.

Program code embodied on a computer readable medium may be transmittedusing any appropriate medium, including but not limited to wireless,wireline, optical fiber cable, RF, etc., or any suitable combination ofthe foregoing.

Computer program code for carrying out operations for aspects of thepresent invention may be written in any combination of one or moreprogramming languages, including an object oriented programming languagesuch as Java, Smalltalk, C++ or the like and conventional proceduralprogramming languages, such as the “C” programming language or similarprogramming languages. The program code may execute entirely on theuser's computer, partly on the user's computer, as a stand-alonesoftware package, partly on the user's computer and partly on a remotecomputer or entirely on the remote computer or server. In the latterscenario, the remote computer may be connected to the user's computerthrough any type of network, including a local area network (LAN) or awide area network (WAN), or the connection may be made to an externalcomputer (for example, through the Internet using an Internet ServiceProvider).

Aspects of the present invention are described below with reference toflowchart illustrations and/or block diagrams of methods, apparatus(systems) and computer program products according to embodiments of theinvention. It will be understood that each block of the flowchartillustrations and/or block diagrams, and combinations of blocks in theflowchart illustrations and/or block diagrams, can be implemented bycomputer program instructions. These computer program instructions maybe provided to a processor of a general purpose computer, specialpurpose computer, or other programmable data processing apparatus toproduce a machine, such that the instructions, which execute via theprocessor of the computer or other programmable data processingapparatus, create means for implementing the functions/acts specified inthe flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computerreadable medium that can direct a computer, other programmable dataprocessing apparatus, or other devices to function in a particularmanner, such that the instructions stored in the computer readablemedium produce an article of manufacture including instructions whichimplement the function/act specified in the flowchart and/or blockdiagram block or blocks.

The computer program instructions may also be loaded onto a computer,other programmable data processing apparatus, or other devices to causea series of operational steps to be performed on the computer, otherprogrammable apparatus or other devices to produce a computerimplemented process such that the instructions which execute on thecomputer or other programmable apparatus provide processes forimplementing the functions/acts specified in the flowchart and/or blockdiagram block or blocks.

The terminology used herein is for the purpose of describing particularembodiments only and is not intended to be limiting of the invention. Asused herein, the singular forms “a”, “an” and “the” are intended toinclude the plural forms as well, unless the context clearly indicatesotherwise. It will be further understood that the terms “comprises”and/or “comprising,” when used in this specification, specify thepresence of stated features, integers, steps, operations, elements,and/or components, but do not preclude the presence or addition of oneor more other features, integers, steps, operations, elements,components, and/or groups thereof.

The corresponding structures, materials, acts, and equivalents of allmeans or step plus function elements in the claims below are intended toinclude any structure, material, or act for performing the function incombination with other claimed elements as specifically claimed. Thedescription of the present invention has been presented for purposes ofillustration and description, but is not intended to be exhaustive orlimited to the invention in the form disclosed. Many modifications andvariations will be apparent to those of ordinary skill in the artwithout departing from the scope and spirit of the invention. Theembodiment was chosen and described in order to best explain theprinciples of the invention and the practical application, and to enableothers of ordinary skill in the art to understand the invention forvarious embodiments with various modifications as are suited to theparticular use contemplated.

REFERENCES

(1) Sinha A, Hripcsak G, Markatou M. Large datasets in biomedicine: adiscussion of salient analytic issues. Journal of the American MedicalInformatics Association 2009; 16(6):759-67.

(2) Singleton J A, Lloyd J C, Mootrey G T, Salive M E, Chen R T. Anoverview of the Vaccine Adverse Event Reporting System (VAERS) as asurveillance system. Vaccine 1999; 17(22):2908-17.

(3) Ambert K H, Cohen A M. A System for Classifying Disease ComorbidityStatus from Medical Discharge Summaries Using Automated Hotspot andNegated Concept Detection. Journal of the American Medical InformaticsAssociation 2009; 16(4):590-5.

(4) Cohen A M. Five-way smoking status classification using texthot-spot identification and error-correcting output codes. Journal ofthe American Medical Informatics Association 2008; 15(1):32-5.

(5) Conway M, Doan S, Kawazoe A, Collier N. Classifying disease outbreakreports using n-grams and semantic features. International Journal ofMedical Informatics 2009; 78(12):47-58.

(6) Farkas R, Szarvas G, Heged s I et al. Semi-automated construction ofdecision rules to predict morbidities from clinical texts. Journal ofthe American Medical Informatics Association 2009; 16(4):601-5.

(7) Mishra N K, Cummo D M, Arnzen J J, Bonander J. A Rule-based Approachfor Identifying Obesity and Its Comorbidities in Medical DischargeSummaries. Journal of the American Medical Informatics Association 2009;16(4):576-9.

(8) Ong M S, Magrabi F, Coiera E. Automated categorisation of clinicalincident reports using statistical text classification. Quality andSafety in Health Care 2010 Aug. 19;doi:10.1136/qshc.2009.036657.

(9) Savova G K, Ogren P V, Duffy P H, Buntrock J D, Chute C G. MayoClinic NLP system for patient smoking status identification. Journal ofthe American Medical Informatics Association 2008; 15(1):25-8.

(10) Solt I, Tikk D, Gal V, Kardkovacs Z T. Semantic classification ofdiseases in discharge summaries using a context-aware rule-basedclassifier. Journal of the American Medical Informatics Association2009; 16(4):580-4.

(11) DeShazo J P, Turner A M. An interactive and user-centered computersystem to predict physician's disease judgments in discharge summaries.Journal of Biomedical Informatics 2010; 43(2):218-23.

(12) Yang H, Spasic I, Keane J A, Nenadic G. A Text Mining Approach tothe Prediction of Disease Status from Clinical Discharge Summaries.Journal of the American Medical Informatics Association 2009 July;16(4):596-600.

(13) Cohen A M, Hersh W R. A survey of current work in biomedical textmining. Briefings in Bioinformatics 2005 March; 6(1):57-71.

(14) Hazlehurst B, Naleway A, Mullooly J. Detecting possible vaccineadverse events in clinical notes of the electronic medical record.Vaccine 2009; 27(14):2077-83.

(15) Melton G B, Hripcsak G. Automated detection of adverse events usingnatural language processing of discharge summaries. Journal of theAmerican Medical Informatics Association 2005; 12(4):448-57.

(16) Murff H J, Forster A J, Peterson J F, Fiskio J M, Heiman H L, BatesD W. Electronically screening discharge summaries for adverse medicalevents. Journal of the American Medical Informatics Association 2003;10(4):339-50.

(17) Wang X, Hripcsak G, Markatou M, Friedman C. Active computerizedpharmacovigilance using natural language processing, statistics, andelectronic health records: a feasibility study. Journal of the AmericanMedical Informatics Association 2009; 16(3):328-37.

(18) Varricchio F, Iskander J, Destefano F et al. Understanding vaccinesafety information from the vaccine adverse event reporting system. ThePediatric Infectious Disease Journal 2004; 23(4):287-94.

(19) Brown E G. Using MedDRA: implications for risk management. DrugSafety 2004; 27(8):591-602.

(20) Bousquet C, Lagier G, Lillo-Le Louet A, Le Beller C, Venot A,Jaulent M C. Appraisal of the MedDRA conceptual structure for describingand grouping adverse drug reactions. Drug Safety 2005; 28(1):19-34.

(21) Bonhoeffer J, Kohl K, Chen Ret al. The Brighton Collaboration:addressing the need for standardized case definitions of adverse eventsfollowing immunization (AEFI). Vaccine 2002; 21(3-4):298-302.

(22) Ruggeberg J U, Gold M S, Bayas J M et al. Anaphylaxis: Casedefinition and guidelines for data collection, analysis, andpresentation of immunization safety data. Vaccine 2007 Aug. 1;25(31):5675-84.

(23) Ewan P W. ABC of allergies: Anaphylaxis. British Medical Journal1998; 316(7142):1442.

(24) Vellozzi C, Broder K R, Haber P et al. Adverse events followinginfluenza A (H1N1) 2009 monovalent vaccines reported to the vaccineadverse events reporting system, United States, Oct. 1, 2009-Jan. 31,2010. Vaccine 2010; 28(45):7248-55.

(25) Quality Investigation of Combo Lot Number A80CA007A of Arepanrix™H1N1 (A503-Adjuvanted H1N1 Pandemic Influenza Vaccine) in Canada.Canadian Ministry of Health; 2010 Mar. 12.

(26) Reblin T. AREPANRIX™ H1N1 Vaccine Authorization for Sale andPost-Market Activities. Canadian Ministry of Health; 2009 Nov. 12.

(27) Uzuner O. Recognizing Obesity and Comorbidities in Sparse Data.Journal of the American Medical Informatics Association 2009;16(4):561-70.

(28) Lewis D D, Ringuette M. A comparison of two learning algorithms fortext categorization. Third Annual Symposium on Document Analysis andInformation Retrieval 1994; 33:81-93.

(29) Carreras X, Marquez L. Boosting trees for anti-spam emailfiltering. 4th International Conference on Recent Advances in NaturalLanguage Processing 2001.

(30) Platt J. Sequential minimal optimization: A fast algorithm fortraining support vector machines. Microsoft Research; 1998. Report No.:MST-TR-98-14.

(31) Hastie T, Tibshirani R, Friedman J. The elements of statisticallearning. Second Edition ed. Springer; 2009.

(32) Stone C J, Hansen M H, Kooperberg C, Truong Y K. Polynomial splinesand their tensor products in extended linear modeling. The Annals ofStatistics 1997; 25(4):1371-425.

(33) Friedman J H. Regularized discriminant analysis. Journal of theAmerican Statistical Association 1989; 84(405):165-75.

(34) Rios G, Zha H. Exploring support vector machines and random forestsfor spam detection. Proceedings of the First Conference on Email andAnti-Spam (CEAS) 2004.

(35) Han E H, Karypis G, Kumar V. Text categorization using weightadjusted k-nearest neighbor classification. Advances in KnowledgeDiscovery and Data Mining 2001; 53-65.

(36) Chang C C, Lin C J. LIBSVM: a library for support vector machines.Department of Computer Science; National Taiwan University; 2001.

(37) Yang Y, Liu X. A re-examination of text categorization methods.:ACM; 1999 p. 42-9.

(38) Friedman M. The use of ranks to avoid the assumption of normalityimplicit in the analysis of variance. Journal of the AmericanStatistical Association 1937; 32(200):675-701.

(39) Yang Y, Pedersen J O. A comparative study on feature selection intext categorization.: Citeseer; 1997 p. 412-20.

(40) Hinrichsen V L, Kruskal B, O'Brien M A, Lieu T A, Platt R. Usingelectronic medical records to enhance detection and reporting of vaccineadverse events. Journal of the American Medical Informatics Association2007 November; 14(6):731-5.

(41) Jha A K, Laguette J, Seger A, Bates D W. Can surveillance systemsidentify and avert adverse drug events? A prospective evaluation of acommercial application. Journal of the American Medical InformaticsAssociation 2008; 15(5):647-53.

(42) Linder J A, Haas J S, Iyer A et al. Secondary use of electronichealth record data: spontaneous triggered adverse drug event reporting.Pharmacoepidemiology and Drug Safety 2010 Oct. 11; 19:1211-5.

(43) Forman G. An extensive empirical study of feature selection metricsfor text classification. The Journal of Machine Learning Research 2003;3:1289-305.

(44) Sebastiani F. Machine learning in automated text categorization.ACM Computing Surveys (CSUR) 2002; 34(1):1-47.

(45) Belkin N J, Croft W B. Information filtering and informationretrieval: two sides of the same coin? Communications of the ACM 1992;35(12):29-38.

(46) Androutsopoulos I, Koutsias J, Chandrinos K V, Spyropoulos C D. Anexperimental comparison of naive Bayesian and keyword-based anti-spamfiltering with personal e-mail messages.: ACM; 2000 p. 160-7.

(47) Bekkerman R, McCallum A, Huang G. Automatic categorization of emailinto folders: Benchmark experiments on Enron and SRI corpora. Center forIntelligent Information Retrieval, Technical Report IR 2004; 418.

(48) Drucker H, Wu D, Vapnik V N. Support vector machines for spamcategorization. IEEE Transactions on Neural networks 1999;10(5):1048-54.

(49) Porter M F. An algorithm for suffix stripping. Program 1980;14(3):130-7.

What is claimed is:
 1. A method, comprising: performing text mining on aset of case reports in text format to determine a set of grammar rulesto be used to determine whether case reports meet a medical condition,the text mining comprising performing feature selection, used todetermine the set of grammar rules, that combines standardized casedefinitions with experience of medical officers for the medicalcondition; and outputting the set of grammar rules.
 2. The method ofclaim 1, wherein the medical condition is anaphylaxis and the casereports are case reports corresponding to flu.
 3. The method of claim 2,wherein the set of case reports is from a vaccine adverse eventreporting system, and the anaphylaxis is caused by a flu vaccine.
 4. Themethod of claim 1, wherein the standardized case definitions compriseBrighton Collaboration criteria.
 5. The method of claim 1, whereinperforming feature selection further comprises: representing importantkeywords as features, the important keywords corresponding to thestandardized case definitions for the medical condition and toinformation determined from the experience of medical officers for themedical condition; determining low level patterns having major and minorcriteria that included one or more of the keywords; and creating highlevel patterns that comprise one or more of the major and minorcriteria.
 6. The method of claim 5, wherein representing importantkeywords as features further comprises creating a list of lemmas torepresent keywords of interest, and the method further comprisesdeveloping a lexicon for the medical condition and building a grammarbased on the lexicon, wherein the building of the grammar forms the setof grammar rules.
 7. The method of claim 6, further comprising using theset of grammar rules to support extraction of the major and minorcriteria and the high level patterns from the case reports.
 8. Themethod of claim 6, wherein the performing text mining is performed at atime prior to a current time and wherein the method further comprises atthe current time applying the grammar rules to one or more new casereports to classify the one or more new case reports as either meetingthe medical condition or not meeting the medical condition.
 9. Themethod of claim 8, further comprising outputting one or more indicationsindicating whether the one or more new case reports are classified asmeeting the medical condition or not meeting the medical condition. 10.The method of claim 8, wherein applying the grammar rules to one or morenew case reports further comprises using a rule-based classifier toapply the grammar rules to the one or more new case reports to classifythe one or more new case reports as either meeting the medical conditionor not meeting the medical condition.
 11. A computer program productcomprising a computer readable storage medium having computer readableprogram code embodied therewith, the computer readable program codeconfigured to cause an apparatus to perform the method of claim
 1. 12.An apparatus, comprising: one or more memories comprising computerreadable code, one or more processors, the one or more processors,responsive to execution of the computer readable code, to cause theapparatus to perform: performing text mining on a set of case reports intext format to determine at least case reports indicated as meeting amedical condition, the text mining comprising applying a set of rulesembodying features selected using both standardized case definitions forthe medical condition and experience of medical officers for the medicalcondition; and outputting at least the case reports being indicated asmeeting the medical condition.
 13. A method, comprising: applying one ormore grammar rules to one or more new case reports, the one or moregrammar rules previously determined at least by performing text miningcomprised of performing feature selection, used to determine the set ofgrammar rules, that combines standardized case definitions withexperience of medical officers for the medical condition; andoutputting, based on the applying, one or more indications of whetherthe one or more new case reports meet or do not meet the medicalcondition.
 14. The method of claim 13, wherein the medical condition isanaphylaxis and the case reports are case reports corresponding to flu.15. The method of claim 14, wherein the set of case reports is from avaccine adverse event reporting system, and the anaphylaxis is caused bya flu vaccine.
 16. The method of claim 13, wherein the standardized casedefinitions comprise Brighton Collaboration criteria.
 17. The method ofclaim 13, wherein performing feature selection further comprises:representing important keywords as features, the important keywordscorresponding to the standardized case definitions for the medicalcondition and to information determined from the experience of medicalofficers for the medical condition; determining low level patternshaving major and minor criteria that included one or more of thekeywords; and creating high level patterns that comprise one or more ofthe major and minor criteria.
 18. The method of claim 17, whereinrepresenting important keywords as features further comprises creating alist of lemmas to represent keywords of interest, and the method furthercomprises developing a lexicon for the medical condition and building agrammar based on the lexicon, wherein the building of the grammar formsthe set of grammar rules.
 19. The method of claim 18, further comprisingusing the set of grammar rules to support extraction of the major andminor criteria and the high level patterns from the case reports. 20.The method of claim 13, wherein applying one or more grammar rules toone or more new case reports further comprises using a rule-basedclassifier to apply the grammar rules to the one or more new casereports to classify the one or more new case reports as either meetingthe medical condition or not meeting the medical condition.
 21. Acomputer program product comprising a computer readable storage mediumhaving computer readable program code embodied therewith, the computerreadable program code configured to cause an apparatus to perform themethod of claim
 13. 22. An apparatus, comprising: one or more memoriescomprising computer readable code, one or more processors, the one ormore processors, responsive to execution of the computer readable code,to cause the apparatus to perform: performing text mining on a set ofcase reports in text format to determine at least case reports indicatedas meeting a medical condition, the text mining comprising applying aset of rules embodying features selected using both standardized casedefinitions for the medical condition and experience of medical officersfor the medical condition; and outputting at least the case reportsbeing indicated as meeting the medical condition.